pkg/iscp/index_sqlwriter.go

• Implements IndexSqlWriter interface with three concrete
implementations: FulltextSqlWriter, IvfflatSqlWriter, and
HnswSqlWriter
• Provides SQL generation for different vector index
algorithms (fulltext, IVFFLAT, HNSW) with CDC operations (insert,
upsert, delete)
• Includes factory function NewIndexSqlWriter to
create appropriate writer based on algorithm type
• Implements generic
type support for HNSW with HnswSqlWriter[T types.RealNumbers]

pkg/vectorindex/hnsw/sync.go

• Implements CDC synchronization functionality for HNSW vector indexes
with CdcSync function
• Provides HnswSync struct for managing index
updates, insertions, and deletions
• Includes parallel processing
support for bulk operations and sequential updates
• Generates SQL
statements for metadata and index table updates

pkg/vectorindex/hnsw/model.go

• Implements HnswModel struct for HNSW vector index operations with
generic type support
• Provides methods for index building, loading,
saving, searching, and CDC operations
• Includes file-based
persistence with chunked loading from database
• Supports concurrent
operations with atomic counters and proper resource management

pkg/iscp/index_consumer.go

• Implemented IndexConsumer for processing index synchronization data

• Handles both snapshot and tail (CDC) data processing modes
• Manages
SQL execution through channels and transaction handling
• Supports
different index algorithms (HNSW, IVF-flat, fulltext)

pkg/sql/compile/ddl.go

• Integrated ISCP job management into DDL operations
• Added CDC task
creation/deletion for index operations
• Updated table operations
(create, drop, truncate, alter) to handle index CDC tasks
• Added
support for async index updates with PITR integration

pkg/sql/compile/cdc_util.go

• Added utility functions for managing CDC tasks and PITR for indexes

• Implements job registration/unregistration with ISCP system
•
Handles creation and deletion of index-specific CDC tasks
• Manages
PITR lifecycle for index synchronization

pkg/vectorindex/types.go

• Added CDC-related data structures and constants
• Implemented
VectorIndexCdc and VectorIndexCdcEntry with generic types
• Added CDC
operation types (INSERT, UPSERT, DELETE)
• Enhanced parameter
structures with async support

pkg/sql/plan/function/func_hnsw.go

• New function hnswCdcUpdate for handling HNSW CDC (Change Data
Capture) updates
• Processes database, table, type, dimension, and CDC
JSON parameters
• Calls hnsw.CdcSync for float32 vector types with
logging

pkg/sql/compile/ddl_index_algo.go

• Added async index support for fulltext indexes with CDC task
creation
• Enhanced HNSW index handling to register CDC update tasks
•
Added logging import and async parameter checking

pkg/vectorindex/sqlexec/sqlexec.go

• Added new RunTxn function for executing transactions with proper
context setup
• Handles account ID extraction and SQL executor
configuration
• Provides transaction execution with proper options and
error handling

pkg/sql/parsers/tree/create.go

• Added Async boolean field to IndexOption struct
• Enhanced Format
method to output "ASYNC " when async flag is true

pkg/sql/plan/function/list_builtIn.go

• Added new HNSW_CDC_UPDATE function definition with proper overload
•
Function accepts 5 parameters including database, table, type,
dimension, and CDC data
• Returns uint64 type and uses hnswCdcUpdate
as execution logic

pkg/sql/plan/function/function_id.go

• Added HNSW_CDC_UPDATE function ID constant (349)
• Updated
FUNCTION_END_NUMBER to 350
• Registered "hnsw_cdc_update" function
name mapping

pkg/sql/parsers/dialect/mysql/keywords.go

• Added "async" keyword mapping to ASYNC token
• Registered new
keyword in the MySQL dialect parser

pkg/iscp/consumer.go

• Added support for ConsumerType_IndexSync consumer type
• Returns
NewIndexConsumer for index synchronization operations

pkg/fulltext/types.go

• Added Async field to FullTextParserParam struct
• Enhanced fulltext
parser parameters to support async operations

pkg/sql/parsers/dialect/mysql/mysql_sql.y

• Added ASYNC token definition and grammar rules
• Enhanced index
option parsing to handle async flag
• Added async keyword to
non-reserved keywords list

pkg/iscp/util.go

• Uncomments and enables support for additional data types in
extractRowFromVector and convertColIntoSql functions
• Adds support
for JSON, bit, array types, date/time types, decimal types, UUID, and
other specialized types
• Includes appendHex function for binary data
formatting
• Improves NULL value handling with proper type casting

pkg/sql/compile/alter.go

• Adds ISCP job cleanup during ALTER TABLE operations
• Includes
DropAllIndexCdcTasks call to remove CDC tasks for temporary tables
•
Adds fulltext index handling in the reindex process
• Improves error
handling and logging for ALTER TABLE copy operations

pkg/vectorindex/hnsw/search.go

• Refactored HnswSearch to use generic types with types.RealNumbers
constraint
• Replaced HnswSearchIndex with HnswModel[T] for better
type safety
• Simplified search implementation by removing file
loading logic
• Updated metadata loading to use generic LoadMetadata
function

pkg/sql/plan/build_dml_util.go

• Added async index support to skip synchronous index operations
•
Updated multi-table index handling to check for async configuration
•
Modified fulltext and IVF index processing to respect async settings
•
Enhanced MultiTableIndex structure with IndexAlgoParams field

pkg/sql/plan/fulltext.go

• Enhanced fulltext index tokenization to support both table and
values scans
• Added support for composite primary keys in fulltext
operations
• Improved parameter handling for different scan types
•
Added primary key type extraction for values-based operations

pkg/catalog/secondary_index_utils.go

• Added async parameter support to index configuration
• Implemented
IsIndexAsync function to check async settings
• Enhanced parameter
parsing to handle async flag
• Updated fulltext and vector index
parameter handling

pkg/sql/plan/build_show_util.go

• Enhanced ConstructCreateTableSQL to handle async parameter from
index algo params
• Added JSON parsing for async flag and appends
"ASYNC" to index string when true
• Improved error handling for JSON
parsing operations

pkg/sql/plan/function/func_cast.go

• Enhanced strToArray function with dimension validation
• Added
bypass for max dimension check and proper error handling
• Improved
array conversion with dimension mismatch detection

pkg/sql/colexec/table_function/hnsw_create.go

• Updated hnswCreateState to use generic HnswBuild[float32] type
•
Modified NewHnswBuild call to use generic float32 type parameter

pkg/sql/plan/hnsw.go

• Commented out TABLE_SCAN node type validation
• Removed strict node
type checking for HNSW query building

pkg/iscp/mock_consumer.go

• Updated context creation to use catalog.System_Account instead of
hardcoded uint32(0)
• Improved system account constant usage for
consistency

pkg/container/types/types.go

• Added DescString method cases for T_array_float32 and
T_array_float64
• Returns formatted vector type descriptions like
"VECF32(128)" and "VECF64(128)"

pkg/sql/colexec/table_function/hnsw_search.go

• Updated newHnswAlgoFn to use generic NewHnswSearch[float32] call
•
Modified HNSW search algorithm instantiation with float32 type
parameter

pkg/iscp/data_retriever.go

• Added GetAccountID() and GetTableID() methods to DataRetrieverImpl
•
Provides access to account and table identifiers for data retrieval
operations

pkg/sql/plan/types.go

• Added IndexAlgoParams field to MultiTableIndex struct
• Enhanced
multi-table index structure to store algorithm parameters

pkg/iscp/types.go

• Added GetAccountID() and GetTableID() methods to DataRetriever
interface
• Extended interface to provide access to account and table
identifiers

test/distributed/cases/vector/vector_hnsw.result

• Updated error message format for vector dimension mismatch
• Changed
from "vector ops between different dimensions" to "expected vector
dimension X != actual dimension Y"

test/distributed/cases/vector/vector_index.result

• Updated error message format for vector dimension validation
•
Improved error message clarity for dimension mismatch scenarios

test/distributed/cases/array/array.result

• Updated vector dimension error messages to use clearer format
•
Changed error text to "expected vector dimension X != actual dimension
Y"

pkg/iscp/index_consumer_test.go

• Adds comprehensive test suite for index consumer functionality
•
Includes mock implementations for data retrieval, SQL execution, and
error handling
• Tests both snapshot and tail data processing
scenarios for HNSW indexes
• Validates SQL generation and CDC
operation handling

pkg/vectorindex/hnsw/sync_test.go

• Added comprehensive test suite for HNSW CDC synchronization
functionality
• Tests cover various scenarios: empty sync, upsert,
delete, mixed operations, and multi-file handling
• Includes mock
functions for SQL execution and streaming operations
• Tests handle
shuffled data and large datasets (up to 1M entries)

pkg/iscp/index_sqlwriter_test.go

• Added test suite for index SQL writers (fulltext, HNSW, IVF-flat)
•
Tests cover insert, upsert, delete operations for different index
types
• Includes tests for composite primary keys and various data
types
• Validates SQL generation for different vector index algorithms

pkg/iscp/util_test.go

• Added utility tests for data type conversion and SQL generation
•
Tests various data types including JSON, arrays, dates, decimals,
UUIDs
• Validates proper SQL formatting for different vector and
scalar types
• Includes comprehensive type conversion validation

pkg/vectorindex/hnsw/search_test.go

• Enhanced existing search tests with multi-file support
• Added mock
functions for 2-file scenarios and catalog operations
• Extended test
coverage for metadata and index batch creation
• Added utility
functions for creating test batches with different file configurations

pkg/vectorindex/hnsw/model_test.go

• Added comprehensive test suite for HNSW model operations
• Tests
model loading, searching, adding/removing vectors, and SQL generation

• Includes error handling tests for nil model scenarios
• Validates
model state management and persistence operations

pkg/sql/plan/function/func_hnsw_test.go

• Added test suite for HNSW CDC update function
• Tests various error
conditions and parameter validation
• Validates null parameter
handling and JSON parsing
• Ensures proper error handling for invalid
inputs

pkg/vectorindex/hnsw/build_test.go

• Updated build tests to use generic HNSW types
• Modified test
functions to work with HnswModel[float32] instead of HnswSearchIndex
•
Updated constructor calls to use generic type parameters
• Maintained
existing test functionality with new type system

pkg/sql/parsers/dialect/mysql/mysql_sql_test.go

• Added test cases for ASYNC keyword in fulltext and vector index
creation statements
• Updated expected output to include ASYNC keyword
in uppercase format

pkg/vectorindex/types_test.go

• New test file for vector index CDC operations
• Tests Insert,
Delete, Upsert operations and JSON serialization
• Validates CDC state
management and JSON output format

test/distributed/cases/vector/vector_ivf_async.result

• Test results for IVF vector index with ASYNC keyword functionality
•
Validates async index creation, data insertion, and vector similarity
queries
• Demonstrates proper async index behavior with sleep delays

test/distributed/cases/vector/vector_ivf_async.sql

• Test cases for IVF vector indexes with ASYNC keyword
• Tests index
creation, data loading, and vector similarity searches
• Includes
sleep statements to allow async operations to complete

test/distributed/cases/vector/vector_hnsw_async.result

• Test results for HNSW vector index with ASYNC functionality
•
Validates async HNSW index creation and vector operations
• Shows
proper handling of CDC updates and similarity searches

test/distributed/cases/fulltext/fulltext_async.sql

• Test cases for fulltext index with ASYNC keyword support
• Tests
async fulltext index creation and search functionality
• Includes
multilingual content and null value handling

test/distributed/cases/fulltext/fulltext_async.result

• Test results for async fulltext index functionality
• Validates
async fulltext search operations and result accuracy
• Shows proper
handling of multilingual and null content

pkg/vectorindex/hnsw/build.go

• Refactors HNSW build functionality to use generic types with
HnswBuild[T types.RealNumbers]
• Removes HnswBuildIndex struct and
related methods (moved to model.go)
• Updates function signatures and
type definitions to support generic vector types
• Maintains
multi-threaded building capabilities with channel-based communication

pkg/sql/plan/function/function_id_test.go

• Adds HNSW_CDC_UPDATE function ID (349) to predefined function IDs
map
• Updates FUNCTION_END_NUMBER from 349 to 350 to accommodate new
function

pkg/iscp/watermark_updater.go

• Fixed bug in unregisterJobsByDBName to handle empty tableIDs array
•
Corrected null check index in queryIndexLog function
• Added
conditional execution to prevent SQL errors

pkg/iscp/iteration.go

• Added error handling for CollectChanges function call
• Enhanced
consumer execution with proper tenant context setup
• Fixed context
propagation for system account operations

pkg/sql/plan/build_ddl.go

• Fixed fulltext index table building to always process
IndexAlgoParams
• Removed conditional check that prevented parameter
processing